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Abstract 

The quality of Education in Chile is a controversial topic that has been in the public debate in the last 
several years. To ensure quality in graduate programs, accreditation is compulsory. The current 
article presents a model to improve the process of self-regulation. The main objective was to design a 
Model of Quality Assurance for Postgraduate Programs in order to constitute a theoretical, 
mathematical, and informatics reference that would optimize the processes of self-regulation, self- 
evaluation, and accreditation of master and doctorate programs from the Universidad Catolica de la 
Santisima Concepcion, Chile. This descriptive research is based on a mixed methods approach. The 
proposal was intended through theoretical and empirical references related to the accreditation 
systems. The analysis process was conducted with key informants, and the informatics instrument 
was created and validated through expert judgment. After the analysis, the model was optimized 
considering the expert’s suggestions. As a result of the optimization process, a matrix of eight 
dimensions was obtained and it is available online in order to be used by the heads of postgraduate 
programs. Finally, a model with four main stages was achieved in order to install a self-regulation and 
a self-evaluated culture that leads to accreditation as evidence of the quality of postgraduate 
programs. 
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Quality Assurance for Postgraduate Programs: Design of a Model 

Applied on a University in Chile 

Discussion of quality benchmarks in postgraduate programs has become highly relevant. Examples 
of this process across the world include the following: Universities in USA are accredited by national 
agencies such as Accreditation Board for Engineering and Technology (ABET), National Architectural 
Accreditation Board (NAAB), Liaison Committee on Medical Education (LCME) and American Bar 
Association (ABA). In Latin America, countries like Brazil and Argentina present experience in the 
accreditation of postgraduate programs, in charge of the Coordenagao de Aperfeigoamento de Pessoal 
de Nivel Superior (CAPES) and Sistema Interuniversitario de Cuarto Nivel (SICUN). In Spain, 
ANECA through the Program Mention gives the excellence and recognition of the countiy’s 
universities. Chile has experience with the National Accreditation Commission, in charge of the 
institutional accreditation of undergraduate and postgraduate programs (Comision Nacional de 
Acreditacion, 2013). 

In general terms, starting an accreditation process involves completing a self-evaluation report. The 
institutions must have a permanent self-regulation process that shows the weaknesses and strengths 
of the programs’ implementation. However, it is convenient to have an established plan to regulate 
their practice. This article addresses the importance of supporting these actions. One example is the 
Postgraduate Direction from the Universidad Catolica de la Santisima Concepcion (UCSC), which has 
developed a model based on theoretical national and international framework to guide the self¬ 
regulation, self-evaluation, and accreditation processes. 

The proposed idea was born from the emerging problem in the Chilean Educational system. This is 
related to the excellent training of people in the different areas of knowledge which are contextualized 
in the new cultural settings. 


Theoretical Framework 

The Concept of Quality in Education 

Quality is considered by the International Organization for Standardization (ISO) in its norm ISO 
9000: 2000 as the degree in which essential features achieve requirements (Cohn, 2002). This 
definition is complemented by Juran & Godfrey (2001), for whom there are two vital definitions of 
quality. First, it means that those product’s characteristics meet customer needs and satisfy them 
(related to income). Second, quality means no deficiencies or absence of errors requiring rework or 
resulting in faulty operation, customer dissatisfaction and/or complaints (oriented to costs). These 
concepts are closely linked to the concept of total quality or excellence or total quality management 
that have been defined as any kind of forms through which the institution meets the needs and 
expectations of its clients, its workers, entities involved financially, and society in general (Santillan 
& Asmat, 2015). 
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The quality of the Chilean education is being treated as a demand that emerges from the social, 
political, and economic system to the educational system. It has been focused on learning 
achievements, results on standardized tests, teachers’ performance, and comparison with standards 
emanating from the international context. This situation is contextualized on the demands that 
emerge from the incorporation of Chile to the countries of the Organization for Economic Co¬ 
operation and Development (OECD, 2013). 

The quality of education enables us to understand it located in structural complexities, cultural 
connotations, and the epistemological, social, and ethical implications. It is intimately linked with the 
academic excellence that can be understood as the quality of ideas, ethical principles and superior 
actions (beyond the minimum required). 

Pedagogically, the quality of learning should be measured not only by the ability to manage 
information, the intellectual and cognitive ability to evoke theoretical concepts, and practical 
experiences recorded in memory, because it might primarily place emphasis on the ability to co¬ 
construct knowledge. At the same time, it is vital to consider the current cultural transition towards 
a fully globalized society, in which students learn in virtual and face-to-face contexts. The skills and 
knowledge might be assessed as intellectual, theoretical, and practical constructs mid by any kind of 
language. The quality of learning should be formalized, as metacognitive and meta-language through 
a communicational ability of students. To assess quality of learning, it is useful to diagnose 
knowledge, competences, skills, and ethical values from the students’ repertoire of previous 
experiences. To educate on quality and based on academic excellence is look at more than 
standardized test weights. It is an explicit distinction between educating for progress, which is 
economic and material development, and educating for development, which is to integrate the human 
and cultural aspects. 

The quality of education is emerging potentially and auspiciously in people who are responsible for a 
fair and equitable society, a distributive and sustainable economy, and a vibrant culture. 

Referents to Create the Model 

Chilean literature review. The Interuniversity Development Center, an international 
non-profit corporation created to promote linkages between leading universities in Latin America and 
Europe, defines quality assurance as a common name for a variety of mechanisms to monitor, ensure, 
and promote the quality of higher education institutions, which contributes to a better understanding 
of these processes as an aspect of progressive development (Centro Interuniversitario de Desarrollo, 
2012). 

In 2006, the law 20.129 established a National System of Quality Assurance in Higher Education and 
defines the Comision Nacional de Acreditacion (CNA, National Accreditation Commission) and the 
advisory committees’ quality assessment. This system is materialized through the implementation of 
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four functions: information, licensing of new institutions, institutional accreditation, and 
accreditation of programs (Pitta, 2013). CNA and advisory committees define the criteria and 
procedures applicable to the accreditation of programs and institutions. The system of quality 
assurance at universities is based on a process of continuous improvement, which can be seen as a 
cyclical process of self-regulation, where the self-evaluation and comparison with standards is a 
fundamental component to design and implement improvement measures required. This self- 
evaluation is contrasted with the observation and expert judgment of external peers. 

In the case of postgraduate programs, self-evaluation is understood as a process in which the active 
community of a particular program (administrators, academics, students and graduates, as well as 
representatives of labour and business sector) critically examines the operation of the program, in 
order to detect their strengths and weaknesses and ensure quality. In this sense, the self-evaluation 
process requires the active participation of all the actors involved in the program and they are fully 
aware of the scope and objectives of the process in which they are involved. 

Regarding the comparison standards, CNA provides only criteria, which define some general 
requirements in the following areas: 

• character and program objectives, 

• admission requirements, 

• programme length, 

• programme structure, 

• thesis procedures (doctoral or master), 

• rating and qualification of teachers of the program, and 

• institutional support for the program. 

The evaluation instruments considered in the process of accreditation of the programs are: 

• a self-evaluation report, 

• a background report, and 

• an external evaluation report. 

The critical point of this process is that since there are no indicators and globally accepted standards, 
self-evaluation is based on the standards and indicators that the institution sets. External evaluators, 
however, have their own mental models about what the desired values are. This can cause major 
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difficulties in this process. For this reason, it is highly desirable that the institution establishes 
standards and generally accepted nationally and internationally indicators. 

International literature review. United Nations Educational, Scientific and Cultural 
Organization (UNESCO, 2009) states that quality standards must reflect all the objectives of higher 
education: especially, to foster in students critical and independent thinking and ability to learn 
throughout life. Moreover, the Accreditation Manual sets clear expectations in the areas of academic 
quality, educational services, continuous improvement, and ethical financial practices (Distance 
Education Accrediting Commission [DEAC], 2016). 

In 2003, the ministers of the signatoiy states of the Bologna Process invited the European Network 
for Quality Assurance in Higher Education (ENQA, 2009) to develop an agreed set of standards, 
procedures, and guidelines for quality assurance. Also they wanted to explore ways to ensure an 
adequate peer review system for quality assurance or agencies or accreditation bodies (or a system 
and agency or accreditation body). These criteria and general guidelines cover all higher education 
and have no special requirements on the postgraduate. European accreditation programs exist in 
some areas of postgraduate study (e.g., MBA) by specialized agencies. 

The quality code in the United Kingdom is the definitive reference for all providers of higher education 
at all levels. In this code, or listing standards, the Quality Assurance Agency sets out what the public 
expects of them. This code was lifted from the expectations of all relevant actors, students, teachers, 
parents, employers, and industiy. It covers the four nations of the UK and all institutions of higher 
education that operate internally or abroad. It was developed primarily to protect the interests of 
students, regardless of whether they are full or part time, or undergraduate or postgraduate. Only B11 
chapter of this code specifically refers to postgraduate programs and establishes 18 specific indicators 
for these programs (QAA, n.d.). The other indicators in the rest of the code refer to all programs at 
both the undergraduate and postgraduate levels. 

In the United States (US) there are so many local, state, and federal agencies that accredit universities 
or careers (more than 60). There is no single system, however, is in the countiy’s culture, widely 
settled the notion of assessment, which translates as though assessment goes beyond that. The 
institutions must be held accountable by the public or private funds spent and the quality of the 
products or services delivered to consumers. There is Commissions Regional Council Accreditation 
(C-RAC), formed as a professional association dedicated to improve education through accreditation 
(El-Khawas, 2007). 

Another feature of the accreditation process in the US is the initiatives arising from the same 
institutions. Accreditation was defined as the process of self-regulation and external peer review 
adopted by the entire educational community. There is no major conceptual difference between the 
assessment of undergraduate and postgraduate level programs, and the emphasis is kept on teaching 
(Middle States Commission on Higher Education, 2009). 
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One of the experiences applied in the Anglo-Saxon context was called Modelling Criteria for 
Measuring and Evaluating the Quality of Education of the University. The document was based on 
the premise that the quality of university education has to be measured by quantitative assessments 
establishing preliminarily defined criteria, depending on the individual indices on the quality of 
training (Hristova & Zhelezarov, 2006). 

The functions of multi-componential indexes, the correlation dependencies between them, and the 
criteria weight coefficients have been studied applying the method of evaluation experts. It suggest 
that the quality of training can be controlled and evaluated according to three different approaches: 

• At the exit, where knowledge, skills, dispositions, and values acquired by students at the time 
of formation (absolute quality assessment) are verified. 

• Added value method, in which the difference between the input and output level is requested. 
This difference is classified on the effectiveness in the process of training and quality of the 
education product. 

• About quality has indirectly in separate parts, elements, and processes, conditions and 
requirements by passing the educational process. 

The assumption of this approach is that if all these answers at most quality requirements then the 
quality will possess a high valuation. 

The measurement of the quality of university education and receiving quantitative estimates of its 
evaluation is one of the conditions necessary for success management and quality improvement as a 
whole. 


Assessment and quality certification of Higher Distance Education. It is clear that 
any methodology for learning should prove useful and positive, and in this sense there are already 
many global initiatives that work for the establishment of standards to certify the quality of e-learning 
projects (Sangra, 2001b; Rubio, 2003). 

In Chile, the wide range in Higher Distance Education (DHE) contrasts with the absence of specific 
regulations and incentives that encourage the development of a range of quality education. While the 
current National System of Quality Assurance of Higher Education establishes generic standards not 
expressly exclude the Higher Distance Education manner, this legislation omits identify and establish 
criteria specifically adapted to the characteristics of an educational model structured on the basis of 
self-learning mediated by technology requirements, flexibility and globalization. 

In this sense it can say that the existing institutions in the field of quality assurance does not provide 
real quality guarantees regarding distance higher education. This situation not only remains valid 
comparative academic results obtained product of this type of education in relation to the traditional 
classroom model, but it is a potential risk to people who choose to be formed through this modality. 


271 



Quality Assurance for Postgraduate Programs: Design of a Model Applied on a University in Chile 
Careaga Butter, Meyer Aguilera, Badilla Quintana, Jimenez Perez, and Sepulveda Valenzuela 


The reality presented in Chile in the area of quality assurance of Higher Distance Education contrasts 
with the significant experiences in the international arena regarding the evaluation and certification 
of the quality of this educational modality. Some initiatives provide an overview and 
recommendations, for example: the study about Quality On the Line, Benchmarks for Success in 
Internet-based Distance Education (The Institute for Higher Education Policy, 2000), and Guidelines 
for the Evaluation of Electronically Offered Degree and Certificate Programs (The Council of Regional 
Accrediting Commissions, 2009). Quality models in online and open education around the globe 
include State of the art and recommendations, from the International Council for Open and Distance 
Education (Ossiannilsson, Williams, Camilleri, & Brown, 2015) and Benchmarking of Virtual 
Campuses, BENVIC Project (Sangra, 2001a), among others. 

Also useful are evaluation and certification models of the quality of Higher Distance Education 
developed by: a) National Accreditation System of Higher Education (SINAES) of Costa Rica; b) 
Distance Education Accrediting Commission (DEAC), United States; and c) Latin American and 
Caribbean Institute for Quality in Higher Distance (CALED) for Latin America and the Caribbean 
Education, through its Project "Virtual Centre for the Development of Quality Standards for Higher 
Distance Education in Latin America and the Caribbean " (CALED, n.d.). 

The CALED model presents a structure consisting of: criteria, sub-criteria, objectives, standards, and 
indicators. In its basic definitions, it declares a quality concept, objectives, management model, and 
an evaluation model, strongly associated with efficiency and effectiveness and whose antecedent is 
the Ibero-American Model of Excellence in Management (Rubio, Aguilar, Massa, Maldonado, and 
Ramirez, 2005). It also establishes a standard definition that promotes continuous improvement and 
has been valued and incorporated by other models in Latin America. 

The Accreditation Manual DEAC established 12 areas of interest in detailing policies, practices, and 
standards that allow for the appropriate comparison and determine the distance between reality and 
standard. Some of these are: Mission, Purpose, and Objectives; Educational Objectives, Curriculum 
and Materials Program; Educational Services; Student Support Services; Satisfaction and Student 
Achievement; Owners, Managers, Heads of Faculty, Administration Committed, reputation of the 
institution; Practices Admission and Registration contracts; Advertising, promotional literature, and 
recruitment of personnel; Financial responsibility; Requirements for Discounts, Returns, and 
Cancellations; Facilities, Equipment, and Supplies; and finally Research and Auto-Improving (DEAC, 
2015). 

SINAES model aims to evaluate the program based on four dimensions: Relationship Context, Means, 
Educational process, and Results. This evaluation model seeks to visualize, in an inclusive manner, 
the main elements of the educational process. This is an environment that contextualizes resources 
or inputs needed for the educational process, the process itself, and results. These elements are called 
dimensions and are evaluated taking into consideration the characteristics of each program. 
Additionally, the model identifies four elements that complement the evaluation: Eligibility criteria, 
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Sustainability criteria of accreditation and improvement of the program, Meta-evaluation, and Re¬ 
accreditation (SINAES, 2011). 

In this context, the problem that arises through the literature review, and in Chile specifically, is that 
there are no reference models that provide dimensions, categories, standards, and indicators to 
ensure the quality of postgraduate programs. Therefore, in the following sections it is going to present 
the general guidelines about the university context where the idea about a Model of Quality Assurance 
for Postgraduate Program was conceived. This model will provide the first step to have a theoretical 
framework, mathematician and computer application, as instrumental to evaluate these programs. 


General Guidelines about the University Context 

The Universidad Catolica de la Santisima Concepcion (UCSC) is located in the Bio Bio Region and it 
is part of the Rectors Council which is an organization that coordinates the academic work of the 
nation, the 25 public and traditional universities are leaders in research and have institutional 
accreditation (Consejo de Rectores de las Universidades Chilenas, N/D). 

The UCSC has 9000 undergraduate students in 52 majors, and 512 Postgraduate students in 22 
master and two doctoral programs. 

The context that determines the vision and the Declaration of the Principles of UCSC, in relation to 
the development of knowledge and contribution to society, can highlight that it is a Catholic 
institution and it has always been recognized as an incomparable creativity and dissemination centre 
of knowledge for the good of humanity. 

The mission of the university is “to promote, from the Bio Bio Region, the generation and application 
of knowledge, contributing to the cultural, economic, scientific, moral, spiritual, and social 
development of the countiy, supported by a comprehensive educational project, of a high quality 
standard” (UCSC, 2012). 

The guiding principles for the formulation of the Model of Quality Assurance for Postgraduate 
Programs (MQAPP) have been obtained from formal document of the university such as: the Strategic 
Development Plan 2012/2016 (UCSC, 2012), the Educational Project of the Institution (UCSC, 2003), 
and the University Curriculum Framework (UCSC, 2010). 

The strategic lines of this plan as they relate to the development of postgraduate programs considered 
for the model are (UCSC, 2012): the university as a leading knowledge generation and training of 
people at the highest academic level centre, and the university as an outstanding service centre in the 
region and major player in the development of the society. 
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Regarding the educational project, the institution emphasizes the need to collect the formation 
demands of high-level specialists who must be able to research and make innovation projects related 
to knowledge generation, scientific, technological, social, and cultural development. Moreover, the 
university considers that postgraduate “is the proper and adequate environment for the development 
of research to the level required by its mission” (UCSC, 2003, p. 4). 

The University Framework defines the curricular guidelines to create new programs and update the 
old ones. It adopts the learning outcomes and competences model (Garcia Retana, 2011) because it is 
pertinent with its component of responsibility and its capacity of integrating the knowledge. 

In this context, the main objective of this research is to describe the design of a model of quality 
assurance for postgraduate programs. 


Methodology 

This research is developed under a pragmatic philosophical paradigm and based on a mixed methods 
approach (Johnson, Onwuegbuzie, & Turner, 2007). Mixed methods represents a set of systematic 
and empirical research and critical processes. It involves the collection and analysis of qualitative and 
quantitative data, as well as their integration and joint discussion, to make inferences from all the 
information gathered and greater understand the phenomena studied. Indeed, a quantitative 
research is oriented towards the description, prediction, and explanation. It is directed towards the 
measurable or observable data. Furthermore, the qualitative research is oriented to the exploration, 
the description, and the understanding of the phenomena (Hernandez Sampieri, Fernandez Collado, 
& Baptista Lucio, 2010). Moreover, it is a cross-sectional research because the information is obtained 
in a unique moment in time. 

The main aim was to design a model of quality assurance for postgraduate programs, by describing a 
theoretical, mathematical, and informatics benchmarks in order to optimize the processes of self¬ 
regulation, self-evaluation, and accreditation of postgraduate programs from Universidad Catolica de 
la Santisima Concepcion (UCSC) in the Bio Bio Region. 

First, a questionnaire was used to gather information. To complete it, the national and international 
state of knowledge and art was considered through academic databases and institutional 
documentation. Second, the informatics instrument considered a matrix with eight dimensions, 48 
categories, 126 standards and 152 indicators for validation. They were made from the theoretical 
design. Third, the validation process was conducted through expert judgment (national and 
international) and users. The specialities of this experts were: informatics specialist, psychologist, 
engineer, teachers, postgraduate and research directors, and vice rectors. Heads from 18 UCSC 
postgraduate programs participated as users and 13 as judges. Programs involved in this process 
were: Doctorate in Education, Master of Science in Education (accredited), Master in Higher 
Education (accredited), Master in Psychology, Master in Mathematics Education, Master in Applied 
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Linguistics, Master in Computer Education and Knowledge Management, Master in Food 
Management, Master in Geotechnical Engineering, Master of Industrial Engineering, Master in 
Marine Ecology, Master of Science in Family Counseling and Family Mediation, Master in Family and 
Community Health, Master in Private Law, Master in Family Law Procedure, Master in Tax and 
Financial Management, Master in Business, and Master in Creative Communication. 

For analysis of the validation process the Delphi method was used, which seeks the consensus of a 
group of experts based on the analysis and reflection of a defined problem (Varela-Ruiz, Diaz-Bravo, 
& Garcia-Duran, 2012). 

The three implemented phases of the Delphi method to validate the model were: 

• Preliminary phase: the basic elements of the model were established through theoretical 
analysis and literature review. 

• Exploratory phase: an analysis of the matrix validation of expert judgment and users was 
made in order to receive feedback from the users and judges. The validation procedure 
consisted of sending a questionnaire to each coordinator to be digitally self-completed. The 
deadline was three weeks for return by e-mail. The data were transferred to an Excel 
spreadsheet for further analysis using descriptive statistics procedures, including analysis of 
mean, standard deviation, minimum, and maximum. The matrix described was sent with a 
letter of invitation and instructions to complete it. Then, a semantic differential scale, with 
values ranging between 1 and 5, was used to evaluate each category and their respective 
indicators, where the opposite adjectives are Rejected (which is equal to the lower acceptance 
level of the claim) and Accepted (which is equal to the highest acceptance level of claim 5). 

• Final phase: The implementation of the model on a computing platform with information 
obtained from the previous analysis. 


Results: A Model of Quality Assurance for Postgraduate Programs 

Main result of this research is related to the development of a model constituted by three elements: 
the references frames, the theoretical and mathematical; and an informatics platform, on which is 
implemented. Furthermore, the validation of expert and user judgment’s result are shown. 

The Theoretical Frame 

This study was a deep analysis of the theoretical and empirical frameworks related to quality 
assurance in Chile and other countries. It was formed by dimensions, categories, standards, and 
indicators. 
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The dimensions are the macro-references that allow structuring the rationality of the model as a 
coherent conceptual base that is able to define the internal and external requirements of the 
university. The dimensions set levels of complexity and derive the same categories as sub-groups and 
the standards and indicators to assess and measure achievements. The dimensions indicate the 
direction of the actions and cover the hallmarks of the whole, as an integrated piece. They are defined 
by the need to bring together the object of knowledge in a single articulated conceptual base. The 
dimensions considered in the model are: Institutional, Curricular, Academic, Research, Management, 
Infrastructure, Services, and Institutional Relationships. 

The categories are hierarchy grades within a particular order. They constitute a second level of 
conceptual deductive grouping that brings coherence to the model in order to identify sub-groups 
within the dimensions. A category can form a philosophical concept, and if it is mixed with other 
referents, allows a first classification of some very large groups, considering all actual or mental beings 
in certain taxonomy or classification of objects, things, or ideas. 

A standard is a generally accepted norm. It can be facts, resulting from a spontaneous process or 
following a formal acceptance. In this case, it is a set of rules that specify the design and 
implementation features that should have a Model of Quality Assurance, with the specific hallmark 
of the UCSC. In the model the standards are detailed, which have been grouped according to the 
dimensions and categories containing them, for the purpose of specifying through qualitative and 
quantitative indicators that can be observed or measured (or both). 

Finally, an indicator is a qualitative or quantitative benchmark that allows evaluating by observation 
or measurement: the realization of a standard. It is a measure that allows warning parameters of 
progress in achieving objectives and targets that provide a simple and reliable means to measure 
achievement, reflect changes connected to an intervention, or to help assess the results of a 
development agency. 

The validation technique is based on a semantic differential scale developed by the experts, and refers 
to establishing the relationship between the standards and categories, setting its relevance, modifying 
it, or discarding it. Eventually, if experts believe that the standard is adequate, they could assign the 
maximum score; otherwise, if they consider that it is inadequate, the standard could receive the 
lowest score. 

Thus, through the validation process, the standards are ratified as elements of the model, are 
eliminated, or are improved, thanks to the flexible design of the model that allows it to incorporate 
new quality standards. 

Mathematical Frame 

The mathematical model is described by a mathematical expression that relates all proposed 
indicators and calculates a value that represents the percentage of compliance with each dimension. 
This percentage is a measure of the quality established by institutional criteria (in this case, the 
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Universidad Catolica de la Santisima Concepcion) and external criteria (in this case, the National 
Accreditation Commission in Chile). 

For each standard corresponding to a category of a dimension, four types of indicators were defined: 
(a) qualitative type, Boolean, (b) qualitative type with 3-5 Likert scale levels, (c) quantitative 
percentage type, and (d) absolute quantitative type. 

Boolean indicators (T/F; 1/0) were used for those standards with exist/not exist quality. For example, 
if there is a Mission defined, it can be true or false, either 1 or o in the encoding. 

Qualitative indicators with Likert items are used when there is an indicator that can have different 
levels and it value is obtained through a standard instrument as a survey. For example, the satisfaction 
level of students for logistics services. 

The quantitative percentage indicators are defined on the basis of a maximum value defined on the 
same indicator. For example, the percentage budget execution according to the plan, it is clear that 
the percentage is calculated on the total planned budget. 

Finally, the absolute quantitative indicators are numbers indicating units or ratios. For example, 
accrediting years obtained or number of publications per senior teacher. 

For each indicator, the expected value is set. This value is estimated considering the evolution in 
recent years (two to five years), improvement plans, or goals set by the accreditation process. 

In order to combine these indicators, they are selected so that the maximum value of the scale, 1 for 
Boolean, 3, 4 or 5 for the qualitative Likert and, 100% for the quantitative percentage, correspond to 
the ideal scenario. Thereafter all indicators are normalized to a decimal value in the range o to 1, with 
1 being the most favourable and o the worst case. 

In the special case of absolute quantitative indicators, there is no ideal value, only an expected value; 
and the highest value does not always correspond to the best result. For example, the indicator 
average number of students per computer. The expected value is 3 and the maximum acceptable 
value is 10 students per computer. The normalization was performed by a linear translation, so that 
3 corresponds to 1 and 10 corresponds to o. 

The formula to normalize scales where there is a minimum or maximum value and an expected value 
is: 


x — Maximum or minimum value 
Expected value — Maximum or minimum value 


Where x is the current value and it is within the following range: 
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Minimum value < x < Expected value 


In the case that there is a maximum value: 


Expected value < x < Maximum value 


These indicators are combined for each of the eight dimensions: Institutional, Curricular, Academic, 
Research, Management, Infrastructure, Services, and Institutional relationships: 



( 1 ) 


Where: 


i = dimension,i = 1 , 2 , 3 , 4 , 5 , 6,7 y 8 

P l = Score obtained in dimension i, expresed as a percentage 
n, = Number of indicators of the i dimension 

Wj = Weight of the j indicator of the i dimension. I Wj = 1 

Ij = Indicator j; 0 < /, < 1 
4 = Compulsory indicators. Boolean (0 o l). 
n c = amount of critical indicators 


The compulsory indicators correspond to the minimum required by the national accreditation, and 
established on the guidelines for the accreditation of graduate programs agency (CNA, 2015), thus 
mandatory by the university itself through its educational project or strategic plan. These indicators 
are Boolean: they exist or not. Equation 1 is automatically o for any compulsory (Ik) indicator having 
a zero value. For those compulsory indicators, either because the CNA or UCSC require them, all of 
them must have the value of one, otherwise the score of the dimension would be zero. 

The weights of each indicator in the first prototype are all 1. The weights can be chosen by an 
appropriate methodology and the priorities of each graduate program (Middle States Commission on 
Higher Education, 2009). 

The model is designed in a flexible way, so in the future it can assign weights to the different indicators 
according to the relevance of each and if those required are not present, the quality value is zero. 

Informatics Implementation 

In order to design and implement the informatics platform an Incremental Prototype Model was 
applied. “It is an engineering methodology which is presented as appropriate to collect progressive 
contributions of users and experts of technological solutions that are designed to meet educational 
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challenges” (Careaga Butter, Badilla Quintana, & Sepulveda Valenzuela, 2014, p.i). It has six phases 
that are recursive, linking prototypes with pilot programs and stages of expansion and routinization 
(see figure 1). 


Pedagogical 

Comunicational 

Methodological 

Tecnological 

Managemnet 



* Comunicational 

* Methodological 
•Tecnological 


Experimental groups 
Control and monitoring 
Inputs to optimize 


Figure 1. Incremental model prototype (Careaga Butter, Badilla Quintana, & Sepulveda Valenzuela, 
2014). 


The platform is in the prototype stage one; incorporating all dimensions, categories, standards, and 
indicators; delivering specific results by category and expressed as a percentage of overall quality 
assurance. The next stage is the development of prototype 2, which will provide a qualitative and 
quantitative analysis to identify critical indicators. 

This platform was designed using PHP and MySQL (free software). It has an account for each master 
and doctoral program that their coordinators can access using their institutional mail and their 
passwords. After, they have to complete the information required. 

The platform design considered different options of answers that are categorized by scales such as: 
binary (0-1), range (1-3), numeric (o-N) and percentage (0-100%). The platform also considers two 
items to answer: the expected value and actual value. The first one refers to the expected result of 
each indicator per program and the second one is the actual result at the time of applying the 
instrument. Individual and total results per dimension are given automatically by the platform (see 
figure 2). 
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Nivel de certidumbre de calidad en la dimension Curricular : 77.5% 


Q Para finalizar los cambios realizados da click en el boton - ubicado al pie del formulario. 

□ Puedes editar aquellos campos que cuenten con el icono Bf en el titulo de la columna. 


Registros por pagina 

10 * 

Filtrar resultados 




Categoria 

Estandar 

Indicador 

or 

Escala Esperado 

Valores 

Bf Actual 

Normalizado 

1 DISEtiO 

CURRICULAR 

DISEtiO FLEXIBLE 

El diseno curricular es 

coherente con el decreto 
#107/2008 

1-3 3 

3 

1 

2 DISEtiO 

CURRICULAR 

DISEtiO LINEAL/CURSOS/ACTIVIDADES 
ACADEMICAS 

El diseno curricular es 

coherente con el decreto 
#107/2008 

1-3 3 

3 

1 


Figure 2. Computer design of the model of postgraduate quality assurance (Careaga & Meyer, 2013). 

Validation of Expert and User Judgment’s Results 

Results are structured according to the eight dimensions of the model. In general, statistical data 
show agreement referring to the acceptance of the dimensions where the mean is 4.5. 

Regarding to the institutional dimension, it was accepted by the majority of the judges and users 
(M=4.3) this means that the model measures the quality that a program should consider in its 
institutional aspects and it can be inferred that the averages obtained meet the requirements for 
approval. At the same time the category named consistency with the educational project has the 
highest level of acceptance (M=4.8) and the category of curricular framework has the lowest level of 
acceptance (M=3.8) (See Table 1). 

Table 1 

Institutional Dimension’s Results 


Category 

M 

SD 

Min 

Max 

Consistency with the educational project 

4-8 

0-39 

4 

5 

Curricular framework 

3-8 

1-5 

1 

5 

Strategic development plan of 
faculty/institute 

the 4.4 

1-34 

1 

5 


The curricular dimension was accepted by the judges and users (M=4.7). This means that it 
contributes to the clarity in the processes of quality assurance for postgraduate programs. Moreover, 
there is no substantial difference in the results shown in Table 2. The curricular design, the mode, 
the admission requirements, and the graduate profile show an average of 5.0: that means that they 
were approved with the highest level of acceptance. Furthermore, the Curriculum type category has 
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the lowest level of acceptance (M=4.o), which is still a positive result and it can be inferred that the 
averages obtained fulfil the requirements for approval (See Table 2). 

Table 2 

Curricular Dimension’s Results 


Category 

M 

SD 

Min 

Max 

Curricular design 

5-0 

0.00 

5-0 

5-0 

Program types 

4-9 

0.25 

4.0 

5-0 

Mode 

5-0 

0.00 

5-0 

5-0 

Admission requirements 

5-0 

0.00 

5-0 

5-0 

Student selection process 

4-8 

0-34 

4.0 

5-0 

Graduate Profile 

5-0 

0.00 

5-0 

5-0 

Graduation Requirements 

4-9 

0.25 

5-0 

5-0 

Educational processes 

4-5 

0.78 

4.0 

5-0 

Flexibility 

4-5 

1.13 

1.0 

5-0 

Curriculum type 

4.0 

1.81 

1.0 

5-0 

Curricular approach 

4-5 

1.20 

1.0 

5-0 

Plans and programs 

4-7 

0.65 

3-0 

5-0 

Undergraduate-postgraduate articulation 

4-5 

0.78 

5-0 

5-0 


Regarding the academic dimension the results obtained shows an average of 4.4 (M=4.4). 
Furthermore, there are important differences in the results shown in Table 3 where two categories, 
mobility and advanced human capital formation, obtained the lowest level of acceptance (M=3.7). 
It means that their four indicators needed improvement but they were accepted. It can be deduced 
that the averages obtained in the rest of categories fulfil the requirements to be endorsed (See Table 
3 ). 
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Table 3 

Academic Dimension’s Results 


Category 

M 

SD 

Min 

Max 

Endowment 

4-8 

0.40 

4.0 

5-0 

Teaching quality 

4-9 

0.25 

4.0 

5-0 

Research capability of teachers 

4-5 

1.09 

1.0 

5-0 

Collaboration networks 

3-9 

1-75 

1.0 

5-0 

Mobility 

3-7 

1.69 

1.0 

5-0 

Retention 

4-8 

0-54 

5 -o 

5-0 

Graduation 

4-9 

0.25 

4.0 

5-0 

Quality monitoring 

4.6 

0.78 

2.0 

5-0 

Advanced human capital formation 

3-7 

1.61 

1.0 

5-0 


According to the findings obtained from the research dimension, it shows an average of 4.2. 
Furthermore, there are important differences in the results shown in Table 4 where the category 
knowledge management (M=2.6) was rejected and its indicators were changed for two new ones. 
Moreover, the postdoctoral category was considered with the lowest level of acceptance because the 
university does not have any postdoctoral program. However, it is important to consider it for future 
programs. On the other hand, the master and medical specialties categories were accepted (M=5.0) 
at the highest level (See Table 4). 
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Table 4 

Research Dimension’s Results 


Category 

M 

SD 

Min 

Max 

Postgraduate integration research 

4-9 

0.25 

4.0 

5-0 

Postdoctoral 

3-0 

1.63 

1.0 

5-0 

Doctoral 

4-7 

0.50 

3-0 

5-0 

Master 

5-0 

0.00 

5-0 

5-0 

Medical specialties 

5-0 

0.00 

5-0 

5-0 

Knowledge management 

2.6 

1.74 

1.0 

5-0 

According to the findings obtained from The Management Dimension, it was accepted by judges and 
users (M=4.9) contributing to the clarity in the processes of quality assurance for postgraduate 
programs. Moreover, there are no important differences in the results shown in Table 5. The self- 
evaluation, self-regulation, accreditation, rules and regulations, and funding categories show an 
average of 5.0, meaning that they were approved with the highest level of acceptance. Furthermore, 
the employability category has the lowest level of acceptance (M=4.4), which is still a positive result 
and it can be inferred that the averages obtained fulfil the requirements for approval (see Table 5). 

Table 5 





Management Dimension’s Results 





Category 

M 

SD 

Min 

Max 

Self-evaluation 

5-0 

0.00 

5-0 

5-0 

Self-regulation 

5-0 

0.00 

5-0 

5-0 

Accreditation 

5-0 

0.00 

5-0 

5-0 

Rules and regulations 

5-0 

0.75 

2.0 

5-0 

Funding 

5-0 

0.00 

5-0 

5-0 

Administration 

4-9 

0.25 

4.0 

1.0 

Employability 

4-4 

1.00 

5-0 

5-0 
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According to the findings obtained from the infrastructure dimension, it was accepted by judges and 
users (M=5.0) contributing to the clarity in the processes of quality assurance for postgraduate 
programs and its categories show an average of 5.0 and 4.9 (see Table 6). 

Table 6 


Infrastructure Dimension’s Results 


Category 

M 

SD 

Min 

Max 

For teaching 

5-0 

0.00 

5-0 

5-0 

For research 

4-9 

0.25 

4.0 

5-0 


Regarding the services dimension, it was accepted by the majority of the judges and users (M=4.6) 
that this means that the model measures the quality that a program should consider in its services 
aspects and it can be inferred that the averages obtained meet the requirements for approval. At the 
same time the categories named library and computer have the highest level of acceptance (M=5.0) 
and the category of student participation events has the lowest level of acceptance (M= 3.9J. Below is 
a table of the averages obtained for each category (see Table 7). 

Table 7 

Services Dimension’s Results 


Category 

M 

SD 

Min 

Max 

Library 

5-0 

0.00 

5-0 

5-0 

Computer 

5-0 

0.00 

5-0 

5-0 

Scholarship systems 

4-4 

i -35 

1.0 

5-0 

Student participation events 

3-9 

1-75 

5-0 

5-0 


Regarding the entailment dimension, it was accepted by the majority of the judges and users (M=4.5) 
that this means that the model measures the quality that a program should consider in its entailment 
aspects and it can be inferred that the averages obtained meet the requirements for approval. 
Furthermore, the category related to with the society has the highest level of acceptance (M=5.0) and 
the categories related to with the company, with the public system, and institutional have the lowest 
level of acceptance (M= 4.3J. Below is a table of the averages obtained for each category (see Table 8). 
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Table 8 

Entailment Dimension’s Results 


Category 

M 

SD 

Min 

Max 

With the society 

5-0 

0.00 

5-0 

5-0 

With the company 

4-3 

i-34 

1.0 

5-0 

With the public system 

4-3 

i-33 

1.0 

5-0 

Institutional 

4-3 

1-33 

1.0 

5-0 


To sum up, the statistical data shows that the dimension and categories’ averages obtained met the 
requirements for approval where the main changes were applied into their indicators. 


The Model of Quality Assurance for Postgraduate Programs 

This model is designed from the result of the theoretical, mathematical, and informatics frame. It is 
a theoretical system, which aims to provide guidance for the process of self-evaluation, self-regulation 
leading to the design and implementation of improvement plans, and processes of accreditation of 
graduate programs (see Figure 3). 
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Figure 3. Model of quality assurance for postgraduate programs (own elaboration). 

The model consists of four main recursive stages: Self-regulation; Improvement; Self-evaluation and 
Accreditation. 

Self-regulation consists of creating excellent conditions to ensure the quality of the postgraduate 
programs by developing permanent processes of revision of the institutional, academic, curricula, 
management dimension consider on each program, in order to improve until obtaining an optimum 
state according to the quality standards that the model defines. In these stages those responsible for 
the curricula and administrative and financial management are in charge of the process and the 
academic staff. They are able to make decisions about the implementation of continuous 
improvement. 

The phases of self-regulation are: 

> to organize the academic staff, which is formed by the head of program and the main staff of 
teachers that give classes and direct theses. The purpose of this committee is to conduct 
academic development, self-regulation processes, and programs’ improvement. 

> to apply the informatics instrument MQAPP including all the relevant information of each 
program. 

The improvement stage consists of developing strategies and implementing courses of action in order 
to consolidate the strengths and overcome the weaknesses of a postgraduate program. The authority 
in charge must determine goals, timelines, responsibilities, and measurable or observable predefined 
evidence. The formalization of this process is done through improvement reports. 

Its phases are: 

> to prepare a report about the main weaknesses of the program and how to improve them. The 
postgraduate direction delivers the format of this report. 

> to create actions to improve and provide a deadline to achieve them. 

At the end of the self-regulation and improvement processes, a checklist must be applied to decide to 
start the next stage. It consists of the 39 most important indicators for the university. 

Self-evaluation stage consists of review processes from the institutional, academic, curricular, and 
management performance of a postgraduate program. An impartial commission of qualified 
academics analyze and criticize this task to draw a contrast between the program level of development 
and pre-established quality standards by public agencies and private agencies. 
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Its phases are: 

> to organize a self-evaluation committee. 

> to prepare the self-evaluation report. 

> to check the final review of the self-evaluation report by the Institutional Self-Evaluation 
Commission (CAI). 

Accreditation stage consists of certifying the quality levels demonstrated by a postgraduate program. 
It is accredited by duly authorized public or private organizations, which are responsible for 
determining whether the self-evaluation process corresponds to the reality of what was developed in 
the program. Then the agency evaluates the number of years of accreditation according to the 
established levels or rejects if the program does not have the minimum expected level of quality. 

Its phases are: To approve the Peer Evaluation, and to prepare the final Self-evaluation Report and 
Accreditation Report. 

Despite that this model was designed for in-person postgraduate programs, it can be applied to long 
distance education programs in its several modalities: e-learning, b-learning, and m-learning. 
Nevertheless it would be necessary to incorporate other specific standards of these modalities, such 
as (a) technology standards (interface design, connectivity, interactivity, and navigability, among 
others); (b) pedagogical standards (new roles, virtual tutoring, curriculum distribution or evaluative 
procedures); and (c) management standards (curriculum management, administrative management, 
financial management, and technology management). 


Conclusions 

The conclusions obtained by this study are consistent with the base of the model. They consider three 
elements: theoretical, mathematics, and technological. The foundation of a MQAPP requires it to be 
set on the basis of a solid conceptual definition about quality, to guide the design and implementation 
of the model. The predominant notion that best fits this model is aligned with overall quality of 
comprehensively training people to perform competently and effectively in reality understood in its 
complexity and the necessary integration and knowledge management for better conditions of life 
and happiness. 

The mathematical model combines a set of indicators of disparate values through standardization, 
giving equal value to all thanks to the choice of the relative weight of each. This value of quality 
(percentage of compliance with the indicators of the dimension) is valid as the choice of indicators 
that was obtained through an appropriate methodology: expert information, survey of teachers, focus 
group, national and international benchmarks, among others. 
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The implementation of the mathematical model was flexible enough to add or delete indicators during 
the prototype stage. Also the equation applied to each dimension is a quality assurance tool that allows 
the academic unit to improve in that dimension to detect those dimensions with low or zero value. In 
this last case is a warning that the unit is not fulfilling some of the minimum requirements for 
accreditation that are requested. In the prototype all weights (e.g., in equation l) are equal to the unit 
used. It is expected in the next stages that the design of a new prototype to improve and define the set 
of weights according to priorities will be established by the university authority. 

The design of a software platform based on incremental prototyping model was perceived to be 
beneficial because it can be applied in real context, subjecting its components to a recursive process 
of design, pilot, validation, and optimization that allows evaluation and optimization. 

Thus, overall, the feedback from experts and users helps to reduce the number of indicators of the 
model. At the end of the first process, the model has 137 indicators. In order to achieve a successful 
model, it is important to generate a conceptual brochure for internal use in order to establish and 
clarify the most important concepts used in the model to unify the different knowledge areas of the 
university. Moreover, the informatics instrument is going to keep the reusable information up in 
order to save working time and it is going to be intelligent to take the decision of going or not to the 
third stage, the self-evaluation. 

These findings strongly support the need to have a model that gives guidelines to manage the self¬ 
regulation processes to achieve quality. However, the participation of each postgraduate program’s 
coordinators is essential because they can recognize the relevant critical points through the 
application and preparation of an improvement plan taking in consideration the potential impact that 
it has in the accreditation of the program. 

Some final reflections to consider about the model for postgraduate programs proposed in this paper 
include the analysis of complementary data, which must be triangulated in order to pass to an 
accreditation stage. The model has a flexible design which could be adapted both the type of program 
and the definition of new standards. Currently the model is being strengthened, taking into 
consideration a comprehensive review the program structure, a curriculum analysis to update the 
graduate profiles, tracking graduates, and consultation to employers. 

Also the model of quality assurance presented could be an interesting contribution, since it can be 
applied in other postgraduate systems, but requires considering and including local adaptations to 
the standards defined in each educational institutions or countries. 

The quality model might be guided by educating ethical and moral people. They should be able to co¬ 
construct knowledge, to integrate disciplines and understanding the complexity of reality. Moreover, 
it is helpful to motivate and to educate people who develop a respectful identity and promote a 
tolerant performance in a society built in interculturality where the notion of education implies 
cultivating the most authentic expressions of knowledge, science, technology, art, sports, and culture. 
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